smash::details Namespace Reference

Functions
template<typename Converter , class Range , std::enable_if_t< std::is_same_v< Range, Particles >||std::is_same_v< Range, ParticleList >, bool > = true>
void	write_in_chunk_impl (const Range &particles, const OutputFormatter< Converter > &formatter, std::function< void(const typename Converter::type &)> write, std::size_t max_buffer_bytes=1 '000 '000 '000)
	Writes particle data in multiple chunks if the total buffer size exceeds a predefined maximum. More...

Function Documentation

write_in_chunk_impl()

template<typename Converter , class Range , std::enable_if_t< std::is_same_v< Range, Particles >||std::is_same_v< Range, ParticleList >, bool > = true>

void smash::details::write_in_chunk_impl	(	const Range &	particles,
		const OutputFormatter< Converter > &	formatter,
		std::function< void(const typename Converter::type &)>	write,
		std::size_t	max_buffer_bytes = 1'000'000'000
	)

Writes particle data in multiple chunks if the total buffer size exceeds a predefined maximum.

This method avoids creating a single excessively large binary buffer when writing many particles at once. Instead, it splits the write into several smaller chunks. This can prevent excessive memory usage and improve stability on systems or filesystems that may have trouble with very large write calls.

The maximum buffer size is currently set to 1 GB (10^9 bytes), but this can be adapted in the future if needed. If the total data size of the particle block is below this threshold, the method simply delegates to the write function which should perform a single write call.

Otherwise, the data is accumulated particle by particle until the buffer reaches the threshold. The buffer is then given to the write function which should flush to disk.

Note

This utility does not strictly belong to this file. At the moment, the objects using it do not have a clean hierarchy that would allow both OscarOutput and BinaryOutput to inherit a shared implementation, hence it lives here temporarily.

Todo:

Once the hierarchy is cleaned up, move this into a common base class that OscarOutput and BinaryOutput inherit from.

Template Parameters

Converter	Converter used by OutputFormatter to produce the buffer type (must define Converter::type).
Range	Container type — enforced to be either Particles or ParticleList.

Parameters

[in]	particles	Container of particles whose particle_line representation is to be written.
[in]	formatter	Formatter responsible for converting particles into the corresponding Converter::type buffer representation.
[in]	write	Callable that receives each filled buffer chunk and performs the actual write to the underlying output (e.g. file).
[in]	max_buffer_bytes	Maximum buffer size in bytes before the accumulated data is flushed via write (default: 1'000'000'000).

Exceptions

std::runtime_error

If the estimated size of a single particle line exceeds half of max_buffer_bytes. In that case, only one particle would fit per chunk, which defeats the purpose of chunked writing, and the caller must increase max_buffer_bytes accordingly.

Definition at line 695 of file outputformatter.h.

                                                {
  if (particles.size() == 0)
    return;
 
  const std::size_t bytes_per_particle =
      formatter.compute_single_size(particles.front());
 
  if (2.0 * bytes_per_particle > max_buffer_bytes) {
    throw std::runtime_error(
        "write_in_chunk_impl: the estimated size of a single particle line "
        "exceeds half of the configured max_buffer_bytes.\n"
        "This effectively means only one particle would fit per chunk, "
        "which defeats the purpose of chunked writing.\n"
        "Increase max_buffer_bytes to at least twice the particle line size "
        "to use this function correctly.");
  }
 
  if (particles.size() * bytes_per_particle <= max_buffer_bytes) {
    write(formatter.particles_data_chunk(particles));
    return;
  }
 
  using Buffer = typename Converter::type;
  Buffer buffer;
  buffer.reserve(max_buffer_bytes);
  std::size_t current_size = 0;
 
  for (const auto& particle : particles) {
    Buffer line = formatter.single_particle_data(particle);
    const std::size_t line_size = line.size();
    if (current_size + line_size > max_buffer_bytes) {
      write(buffer);
      buffer.clear();
      current_size = 0;
    }
    buffer.insert(buffer.end(), std::make_move_iterator(line.begin()),
                  std::make_move_iterator(line.end()));
    current_size += line_size;
  }
 
  if (!buffer.empty()) {
    write(buffer);
  }
}